A Central Processing Unit (CPU) typically includes an Instruction Fetch Unit (IFU) that is configured to manage the fetching of program instructions from memory. In multi-threaded processors, the IFU is configured to concurrently handle the fetching of program instructions for a number of different threads.
An IFU may include one or more levels of instruction cache, each level of which are small hardware-managed memories that store a subset of program instructions and that can be accessed faster than Main Memory (usually, the first level can be accessed in a single processor cycle).
An IFU may also comprise an on-chip high-speed local memory, such as an Instruction Scratchpad Random-Access Memory (ISPRAM) which is managed in software, either by the programmer or through compiler support. This memory may be used to store critical blocks of code that need to be retrieved with a small and predictable latency. Usually, large ISPRAM arrays are necessary to meet an application's demands. However, integration of such a large memory in a high performance processor can severely damage performance if not handled cleverly, as its access may require several cycles.